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Research on Business Services Automation 


Research Objective: 


We aim to assess the current and long-term effects of business services automation on client 
organizations. While using software to automate/augment work is not a new idea, recent interest 
in service automation has certainly escalated with the introduction of new technologies including 
Robotic Process Automation (RPA) and Cognitive Automation (CA) tools. Many potential 
adopters of the new types of service automation tools remain skeptical about the claims 
surrounding its promised business value. Potential adopters need exposure to actual and realistic 
client adoption stories. Mature adopters want to learn about advanced practices. Academic 
researchers can help educate potential and mature adopters by objectively researching actual 
RPA and CA implementations in client firms, by assessing what the software can and cannot yet 
do, and by extracting lessons on realizing its value. 
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Reimagining the University at Deakin: 
An IBM Watson Automation Journey 


“Deakin University offers a personalized experience, enhanced by innovative digital 
engagement. We lead by creating opportunities to live and work in a connected, 
evolving world.” — Deakin University Website’ 


“LIVE the Future essentially distills down to this: we don’t care where a student is 
geographically, we care where they are academically and helping each student 
succeed during his or her entire journey.” — Professor Jane den Hollander AO, 
Vice Chancellor for Deakin University 


“Digital is the way we live.” — Professor Beverley Oliver, Deputy Vice-Chancellor 
(Education) for Deakin University 


Introduction 


Nearly 40 Australian universities compete for national and international students in a country of 
24 million people. In this paper, we examine how one public university, Deakin University, 
competes for students by reimagining higher education through it’s “LIVE the Future” Vision. 
The vision is enabled by aggressive investments in digital technologies that enhance the 
student experience. Deakin University’s adoption of IBM Watson is one such investment, and 
the focus of this paper.’ We focus on this particular investment because there are few visible 
adopters of cognitive automation (CA) technologies beyond the widely covered IBM Watson 
applications in healthcare. Potential adopters of IBM Watson and other CA technologies need 
realistic examples of the effort required to gain value from such investments. 


Despite being located in what might seem to be “distant” Australia from a global perspective 
(see Sidebar: About Deakin University), the university's innovativeness has brought it 
international attention, and has grown enrollments to 54,000 students by 2016. With its roots in 
and reputation for quality distance education, a third of its student body studies exclusively 
online. Deakin University’s administrators think that online enrollments can increase 
substantially over the next ten years, provided the university delivers an exceptional student 
experience, significantly enabled by digital services. 


This paper first explains the university's vision and the role IBM Watson played within that larger 
picture. Then, we describe the university’s entire adoption journey from conception and 
deployment to future plans. We document the ‘triple-win’ value the investment yielded for the 
university, students, and staff. 


' Our other working papers focus on Deakin’s broader digital strategy and investments. 
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SIDEBAR: About Deakin University 


Deakin University is a public university in the Australian state of Victoria. Established in 
1974, it had over 53,000 students in 2016” across its four physical campuses in Melbourne, 
Geelong, Warrnambool and Burwood.’ The university calls its “Cloud Campus” its fifth 
campus. About 1/3 of its students study solely online.* Deakin prides itself on its overall 
student satisfaction score, which was rated the highest among Victorian universities for six 
consecutive years. Deakin was also rated Victoria’s top ranked university for students under 
50 years of age for the past two years.” 


Finally, we offer key lessons on achieving that value: 


1. Strategy drives technology investments 

2. Manage expectations up, down, across, and out 

3. Don’t under-estimate the data challenge 

4. Treat cognitive technologies as lifelong learners 

5. Supervise all new learning 

6. Educate and continually engage subject matter experts 

7. Expect technical challenges as a first-mover 

8. Negotiate the optimal level of client-provider transparency 

9. Put in place a strong in-house team 

10. Continually innovate because today’s “cool” is tomorrow’s “yawn” 


“LIVE the Future” 


Professor Jane den Hollander, the Vice-Chancellor, is credited as the architect of the “LIVE the 
future” vision, a strategy that aims to put Deakin University at the edge of the digital frontier in 
higher education.° The Vice Chancellor, however, will be the first person to credit Deakin 
University’s entire community for the plan. When she took the position in 2010, she was an 
outsider to Deakin and engaged stakeholders to develop the new vision. She said, “When | 
arrived, | know we needed a new plan because the university was shutting down its distance 
learning. We assembled all the staff and invited everybody to answer the question: What 
should we do next?” The university community settled on “L-I-V-E the Future” through 
Learning, Ideas, Value and Experience. Professor Beverley Oliver, Deputy Vice-Chancellor 
(Education), further explained, “Underneath those words we have four large plans: student 
learning experience, research and development, community engagement and 
internationalization. It’s about giving people a brilliant education, where they’re at physically, 
academically, philosophically and where they want to go in life.” 


Focusing on the student experience part of the vision, the university creates effective and 
personalized digital experiences. Lucy Schulz, the Director of Cloud Campus, explained, “The 
vision of the student journey program is to bring students to the centre of our thinking—in every 
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area and on every level—so that students are enabled to be successful and feel supported 
throughout their time at Deakin.”’ 


Deakin University invests in technological innovations that enhance the student experience. 
William Confalonieri, Chief Digital Officer for Deakin University, explained that technology 
investments are informed by the student culture, which he describes as “the age of impatience” 
— students want digital technologies that are always on, always easy to use, and always fast.* 
Five megatrends inform the university’s eStrategy: Place, Pace, Face, Space, and Trace, which 
corresponds to mobility, flexibility, personalization, collaboration, and information. ° All of 
Deakin’s technology investments, ranging from its learning management system to collaboration 
tools, are orchestrated under one seamless cloud-based hub called DeakinSync.'” DeakinSync 
is the one-stop personalized dashboard that aggregates essential information for every student. 
When Deakin University decided to invest in IBM Watson, it was within the context of further 
enhancing the student experience and integrating the tool into the cloud-based hub. 


Deakin University’s IBM Watson Adoption Journey 


As citizens of the world, many of Deakin University’s administrators, staff and students were 
already familiar with IBM Watson from its televised championship on the game show, Jeopardy! 
Since that event happened back in 2011, it is worth recapping Watson’s performance. 
Jeopardy! is a US game show where humans compete to ask trivia questions after they are 
presented with convoluted clues. Clues contain innuendos, double entendres, metaphors, and 
puns. The game is extremely complex; it requires fast (under a second) processing of 
unstructured, culturally rich, natural language. IBM had to find a new way to program and 
subsequently developed its proprietary supervised machine learning algorithms. After four years 
in development, IBM Watson was tested live on television. Over the course of three days in 
February of 2011, Watson competed against the reigning human champions, Brad Rutter and 
Ken Jennings. '' Watson won, earning $1 million dollars. This bellwether event signaled that the 
age of cognitive automation was upon us. For what other applications might Watson be used? 


From 2011 to 2014, IBM Watson had primarily been deployed in healthcare. No university had 
bought the technology for any other application. In mid-2014, IBM personnel visited Deakin 
University’s Vice Chancellor and Deputy Vice Chancellor to show them Watson. The 
administrators could immediately see the potential value, but they needed to learn more. They 
engaged IBM to do a cognitive value assessment. After that exercise, the administrators 
concluded there was enough potential value to pilot the new technology within a limited domain 
of responding to common asked questions using Watson’s Engagement Advisor application. 


Deakin University adopts Watson 


In October of 2014, Deakin University officially announced that it would adopt Watson. The aim 
was to provide students with a single source of accurate, current, and relevant information 
available on any device.” The university wanted Watson to improve the student experience by 
tailoring a student’s queries to their own student profile. The Vice Chancellor summarized the 
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vision as personalizing the student experience by providing advice and information “just in time, 
just for me, anytime, anywhere, on any device.”” 


IBM was the technical lead and Deakin University was the business lead. The university’s IT 
department was also involved in various technical activities such as linking Watson to Deakin’s 
website. At that time in 2014, there were very few Watson experts in Australia, so IBM sent a 
team from the United States to launch the development. 


Project Development 


By December 2014, Deakin University and IBM agreed to a plan with three releases targeted for 
February, June, and September of 2015. The first release would provide answers to commonly 
asked questions by incoming students, with access provided through its cloud-based hub, 
DeakinSync. The second release would expand Watson’s range of question categories, with 
the aspiration to make Watson a comprehensive destination for student queries. The third 
release aimed to personalize and contextualize the answers. The first release was divided into 
three stages: 


Release 1, stage 1: Collect Questions. The first stage required collecting students’ questions. 
The Deakin University team gathered nearly 20,000 questions from staff and administrators in 
charge of recruiting, scheduling, counseling, advising, and orientation. The IBM team helped to 
categorize the questions by student “intent”, as many questions were essentially seeking the 
same information, even though they were worded quite differently. From the initial collection of 
questions, 2,000 questions were selected for its first release. 


Release 1, stage 2: Find correct answers. The second stage of the first release required 
getting the correct answers to each of the 2,000 questions. The content could come from many 
sources: verbal answers from experts, written responses contained in emails and documents, or 
multi-media content posted on webpages. As many first time CA adopters often discover, 
sources were not always up to date or accurate. The university had to assign a single content 
owner responsible for each subject area and have them provide the correct answers. For 
Watson’s first public release, all the answers were handwritten to ensure accuracy and 
appropriateness with the help of 100 content owners from across campus. 


Release 1, stage 3: Ready content for Watson ingestion. Once the correct answers were 
identified, the third stage entailed “content uplift’, where answers were appropriately worded 
and structured for Watson ingestion. Watson was ready to be tested. 


The university asked students to volunteer during their break to test and further train Watson. 
Over 200 students volunteered. Students indicated if Watson’s answers were correct, 
incomplete, or inaccurate. Their feedback was incorporated into the application to improve 
Watson’s performance (see Figure 1). 
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Figure 1: Student volunteers help train Watson in preparation for launch 
Source: https:/Awww.youtube.com/watch?v=MK9gakgPDoc 


Release 1 Launch 


Deakin University took just four months to get Watson ready for launch in time for student 
orientation week. During orientation week, the Watson team focused on creating awareness of 
Watson with new students. They erected a booth at orientation and showed students what 
Watson was and how to use it. According to one student who worked at the booth, “We were 
inundated by student interest.” * The Watson developers explained that Watson was still 
learning and that the university needed each student’s help to further train Watson. The 
university launched a marketing campaign with the slogan, “I’m helping train Watson” to engage 
students and staff (see Figure 2). 

Students and staff were told, “the more you use Watson, the better it will get at helping you”.” 
This campaign served to temper users’ expectations of Watson’s initial performance. Besides 
the booth at orientation, students were made aware of Watson on Deakin’s website (see Figure 
3) and on the current student webpage portal (see Figure 4). Students access Watson through 
DeakinSync by signing on with their logon ID and password. Although Watson is designed 
primarily for students, Deakin staff members were encouraged to use it for their own enquiries. 
Additionally, about 100 staff members became content owners, responsible for Watson’s 
content going forward. 
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Figure 2: Recruiting new students to train Watson at Orientation Week 
Source: https:/Awww.youtube.com/watch?v=MK9gakgPDoc 


UNIVERSITY 


IBM Watson 


Deakin was the first university in the world to partner with IBM's 
groundbreaking machine learning and artificial intelligence program, 
Watson. Learn how Watson can help you during your time at university. 


Ask and you shall Looking for your unit guide? Food on campus? Study support? 


receive 
Watson's groundbreaking artificial intelligence can give you important information, like 
who you can talk to about your course, how you can fix an enrolment or timetable issue, 
even how to get involved in life at Deakin. 


Better yet, Watson is 24/7, communicates like we do and is constantly learning! 


“Deakin is a young, inspiring and innovative university with great 
ambitions; Watson is a breakthrough and cutting-edge system - it's a 
perfect match and represents thinking right on the edge of the digital 
frontier.” 


WILLIAM CONFALONIERI 
CHIEF DIGITAL OFFICER 


Figure 3: Deakin University’s web portal explaining Watson to students 
Source: http://Awww.deakin.edu.au/life-at-deakin/why-study-at-deakin/ibm-watson 
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Figure 4: Student access to Watson through DeakinSync 
Source: http://www.deakin.edu.au/students 


Watson’s Initial Performance 


Watson answered over 55,000 questions during the first twelve months. The university 
anticipated that students would most frequently ask questions about educational processes, 
such as how to enroll in classes. In reality, students most frequently asked Watson for 
information about finding dates, finding food, and the location of course materials—in that 
order. Common questions, however, do change over the course of a semester. For example, 
questions about finding classrooms are more frequently asked at the beginning of the semester; 
questions about exams are more frequently asked later in the semester. 


Deakin University tracked Watson’s response accuracy very closely, counting the number of 
direct questions Watson answered correctly or incorrectly, the options Watson generated 
appropriately or inappropriately, and the number of queries Watson indicated it did not know 
how to answer (see Figure 5). Watson performed quite well, correctly answering direct 
questions or offering appropriate options about 80 percent of the time. 
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Figure 5: Percentage of questions Watson did not initially understand 
Source: Deakin University presentation, March 2015 


In addition to monitoring Watson’s logs of actual conversations to assess performance, Deakin’s 
Watson team members also looked at students’ ratings of Watson’s performance. After each 
conversation, Watson prompts the student to rate the quality of Watson’s responses (see Figure 
6). Based on a sample of 1,130 feedback ratings, students gave Watson “good to excellent” 
ratings for 63 percent of the queries. This feedback was used to improve performance over 
time. 


Help improve Watson 


1. Rate your experience 


Poor Excellent 


Te) 16 


2. Tell us about your experience (comments are anonymous) 


Submit Feedback 


Powered by 
Opinio Survey Software 


Figure 6: User feedback form for Watson 
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Releases 2 and 3: Expanded Capabilities 


Deakin University continued to expand Watson’s capabilities through its second and third 
releases. By November 2015, Watson was connected to Deakin’s website and online handbook 
to find more answers, was further programmed to personalize information based on campus 
and student type (domestic vs. international), and started to share its confidence ratings for its 
answers with students.” As of 2017, Watson can guide students through common processes 
like submitting assignments, paying for parking, and re-enrolling in study. Watson has been 
trained to answer 6,000 questions (see Appendix for sample conversations with Watson). 
Having concluded the IBM Watson adoption journey up to 2017, we next discuss the value 
generated from the investment. In addition, we outline subsequent developments, such as 
Deakin Genie, that emanated from the learning gained during the Watson project. 


Case Discussion: Achieving the “Triple-win” 


Deakin’s adoption of cognitive automation delivered value to three major stakeholders: The 
University as an institution, students, and staff (see Figure 7). Deakin University is not alone in 
achieving such results. Across our cognitive automation cases, we have called the realization of 
multiple sources of value the “triple-win” of service automation." 


INSTITUTIONAL 


VALUE 


A Improved competitive Round the clock service & Learned new skills 
positioning delivery 4 Focus on more critical 

“ Raised brand awareness & Faster access to critical tasks 

4 Increased personal attention human assistance without ® Help deal with demand 
at scale to support aggressive divulging personal details to volatility 
growth strategy another human being & Raised awareness of the 

& Better content governance Æ Learned new skills potential of cognitive 

Created a single point of truth à Increased response accuracy service automation in 

© Established a repository of = Multi-channel delivery teaching and research 
questions students actually & Perceived investment as 
ask as input to business university commitment 
intelligence to growth 


Figure 7: The triple win for automation at Deakin 


We note also that, as a fourth stakeholder, IBM’s Watson team received value in terms of 
learning about developing and implementing their tools in specific contexts as was the case 
here with tertiary education. 
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Institutional value 


“What were the outcomes? The main value was reputation and profile building 
for the university. It was worth the investment.” — Professor Beverley Oliver, 
Deputy Vice-Chancellor (Education) for Deakin University 


Deakin University was the first university adopter of IBM’s Watson for student services, which 
gave the university worldwide media attention (see Figure 8). Its Watson application won 
prestigious awards. For example, Deakin University earned 1*' place for the global “Wharton-QS 
Re-imagining Education Stars” award within the ICT Support and Services category in 2015.” 
Requests for interviews, site visits, and speaking engagements came pouring in from across the 
globe. For instance, Deakin’s Vice-Chancellor spoke at the 2015 IBM World of Watson as a 
keynote speaker.” The media attention, awards, and public events delivered institutional value 
in terms of improved competitive positioning and raising brand awareness. 


h m VIRTUAL STRATEGYMAGAZINE = M watar New York end five [EE —_ a n Deakin University students to get 
=~ ~ me x ressa answers using IBM’s Watson 
z It wow 8 a 


5 bi ne Deakin University set to j . NETWORKWORLD 
z pen OUPUTER WORLD 
Sala) a al coe © sis De Sn 
ney E 2 CNBC ; i Dr IOTO 
[= eee vil | BUSINESS | °° ~ EAE BUSINESS src 
ao ooo ws aS ok —— 
Deakin University to use machine leaming pm SNEWS = = t a( K ) 
K id n 4 


IBM's Watson sapercompater signs o Asse customers 


IT News Online =o GlobeAdvisor 
a — ee Km Imm PRN Fen we 
ANZ, Deakin University set to 
Korneti WS menee ne erea e s tips Eek es 
vot economia meea penaas 
IBM Watson Areekerates Ghibal Expansion 
iee = = N roe TECH News 
iy 


BUSINESS SB Sree 


ii ee NZ, Deakin University set to harness 
= pe BUSINES "Nac 
Accelerates CRM DAILY 


IBM's Watson picks up first Global E. 
Australian customers Deakin University students to get sack hein dete 
Natson et soo help Deakn Urveratystaderts and ANZ answers using IBM's Watson one 
ree ems r shin paia e 
AN E a <a A 
IBM Goes Global with Increasing | neusencron fa IBM supercomputers to 
Number of Watson Clients launchin Australia -m 


Figure 8: Sample media coverage of Deakin University’s Watson adoption” 
Source: Deakin University presentation, April 22 2015 


Deakin University also gained value from its Watson investment in terms of progressing towards 
its goal of “personal attention at scale”. University administrators aspire to double or even 
triple enrollments over the next decade, but scalability cannot cannibalize its rich student 
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journeys. The administrators see Watson as enhancing the student experience while at the 
same time improving scalability. 


Deakin University reported an unexpected source of value from its Watson investment: better 
content governance. In the process of curating content for Watson, Deakin University 
discovered that many of its data sources reported outdated or inaccurate content. So for 
example, many sources reported conflicting library operating hours. The university remedied 
this by identifying content owners responsible for “a single point of the truth’. Now, the 
content owners post the correct data on their own webpages and any other departments that 
want to include that content within their own domain webpages must point to the original source 
rather than copy and paste content. So for example, the library is the sole content owner for 
library hours and other sources should point to the library's webpage rather than copy the 
content. Professor Jane den Hollander, Vice Chancellor for Deakin University, explained the 
value of a single point of truth, “You go to Deakin, you ask a question, and it’s always 
answered.” Deakin also experienced another benefit from establishing a repository of questions 
students actually asked: better business intelligence. 


To present a balanced view on institutional value, we also report that Deakin University’s 
investment did not produce measurable operational efficiencies or returns-on-investment (ROI) 
in the short-term. Indeed, Gartner reported on Deakin University’s ROI in 2016: “The university 
does not yet have hard return on investment (ROI) metrics for the Watson deployment. 
Attributing ROI budget benefits in terms of reputation and student satisfaction will likely remain 
hard to quantify. Part of the reason for this is the difficulty of attributing benefits to individual IT 
systems such as Watson.” This lack of measurable ROI was a common finding across our 
other CA adopters. In the lessons learned section, we answer the question: How does an 
organization calculate a return on investment when no staff is laid off as a consequence of 
automation? 


Student Value 


“Watson revolutionizes the student experience and engagement and transforms 
the way students get advice and answers to questions about their study and life 
at Deakin.” 


Deakin University reported that IBM Watson yielded multiple sources of student value, including 
round the clock service delivery, faster access to critical human assistance, enhanced skills, 
increased response accuracy, and multi-channel delivery. 


Deakin University aimed to use cognitive automation to provide round the clock service 
availability and delivery. Prior to Watson, students were contacting the university’s offices at 
any time of the day or night and on weekends to ask questions, but most offices closed at 
4:00pm local time and only operated on weekdays. Students were forced to leave voice 
messages or send emails when the offices were closed. If a student left a message after hours 
on a Friday, it could take three days for the student to receive a response. Professor Jane den 
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Hollander, the Vice Chancellor, explained, “Our counselors would come in the morning and 
immediately start dealing with 73 voicemail messages.” As Deakin University increasingly 
expanded its online programs to students around the globe, 24-hour service availability became 
critical. The Vice Chancellor continued, “As our students started to come from everywhere, we 
knew our big vulnerability at the digital frontier was that we couldn’t service them at 24/7. 
Watson is always up and running, making services available to students at anytime.” 


Watson also provides fast access to critical human assistance. Student depression is a 
concern on every university campus. Deakin University has professional counseling services 
available for any student. Its website states, “If there’s something bothering you, however big or 
small, help is at hand. Our counselors are registered psychologists and social workers. They 
have extensive experience working with students with mental health issues, ranging from 
adjustment stress to common mental illnesses like anxiety and depression. The service is free 
and confidential.” Students don’t always have the courage to contact the counseling office 
directly, but some students have confided their despair to Watson. Watson is programmed to 
point students to critical human assistance and it proactively alerts the counseling staff that a 
student needs help. Professor Jane den Hollander, the Vice Chancellor, explained, “We've 
stopped more than a couple of people from spiraling down into depression through urgent 
intervention. Students know that when we are concerned for them, we have someone 
confidentially contact them.” 


Hundreds of students were involved in the training and testing of Watson to ensure Watson 
understood the student voice. Professor Jane den Hollander, Vice Chancellor for Deakin 
University, explained, “I wanted students involved because | thought who’s going to be clever 
with this technology? - it will be the students. So we got the students engaged and they thought 
the technology was cool. They loved the idea of training a machine. They understood quickly 
that the more they interacted with the machine, the better the machine performed. There were 
some very smart students who led all that.” Student engagement proved very valuable, not only 
in enabling the first release to be delivered on time, but it gave students the opportunity to learn 
new skills while developing a new leading-edge digital technology.” 


The students recruited for Watson’s content curation, testing, and training learned valuable new 
skills about the technology. Additionally, their involvement also helped to hone more general 
skills; Deakin University has eight specific learning outcomes for all students, regardless of 
major. There are: (1) digital literacy, (2) communication skills, (3) critical thinking, (4) problem 
solving, (5) discipline-specific knowledge, (6) self-management, (7) teamwork and (8) global 
citizenship. The university believes these skills are highly valued by employers and prepare 
students to be work-ready. The students involved in the Watson project had to work on a team 
to quickly solve problems to meet a tough deadline of just four months! They also had to help 
disseminate knowledge to the entire student body. 


Students also gained value from the university’s better content governance and single point of 
truth: they now get more accurate responses to their questions. Prior to Watson, phone 
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contact and email inquiries were the main channels for students’ questions. Watson opened 
another channel for service delivery. 


Staff Value 


Watson objective: “Free up time for student service staff to enable them to respond 
and attend to more critical and complex issues. ”” 


Given that automation technologies can threaten human jobs, one might naturally assume that 
the staff would feel threatened by the technology. Deakin University initially faced a small 
amount of apprehension that was quickly overcome when the university told them the purpose 
of Watson was to help them, not eliminate them. Professor Beverley Oliver, Deputy Vice- 
Chancellor (Education), explained, “We told the narrative very carefully when we introduced 
Watson because we didn’t want people to worry: Am | going to lose my job?” The university 
was very careful about messaging the intent and purpose of Watson as an alternative channel 
and not as staff replacement. 


Some value to staff members was evident. The staff members who serve as content owners 
learned new skills in curating content for Watson ingestion and ongoing upkeep. Watson also 
freed up staff for higher-value student support, which was a main objective from the start. 
Watson also helped to deal with demand volatility, as students’ service needs are not uniform 
over a semester. As of 2017, Watson answered about 3,000 queries per week—questions the 
staff did not have to answer. We note, however, that as enrollments increased, all channels 
were experiencing increased volumes, so while the staff was answering fewer common 
questions, they were no less busy. 


The Watson project also made staff aware of the potential of cognitive automation in 
teaching and research in addition to student services. And certainly, the staff perceived the 
investment as evidence of the university’s commitments to growth and quality. 


How did Deakin University deliver value to the institution, students, and staff? The university 
enacted a number of practices that serve as lessons for other organizations. 


Lessons Learned 


As an early adopter of cognitive technologies, Deakin University’s case study offers a number of 
insights for other organizations considering similar technologies. Given the university’s 
ambitions to reimagine higher education and its subsequent adoption of the most formidable of 
all cognitive tools—IBM Watson—the learning points may not apply to organizations seeking 
more modest aims. Where possible, we supplement Deakin University’s lessons with lessons 
from our other case studies. 
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1. Strategy drives technology investments 


“We don’t disrupt for the fun of it—that would be reckless. We disrupt to solve a 
problem.” — Professor Beverley Oliver, Deputy Vice-Chancellor (Education) for 
Deakin University. 


Although this report focused on Deakin University’s use of IBM Watson, it is vital to understand 
that strategy is driving technology investments. Deakin University does not have a “Watson” 
strategy; rather it has an international growth strategy focused on enhancing the student journey 
through its “LIVE the Future” vision. Watson just happened to be one among many investments 
that enable the strategy. 


This lesson was also evident in our study of other organizations that achieved the triple-win for 
shareholders, customers, and employees from service automation.” In our research, “digital 
transformation” was a common C-suite strategy. Digital strategies aimed to ease customer 
journeys from initial prospecting through to account set up, ordering, delivering, receiving, 
maintaining, and paying for products and services seamlessly—a digital assembly line if you 
will. 


In contrast to organizations that achieved the triple-win, several of our less successful cases 
became enthralled with automation technologies and bought software licenses or started 
building bespoke systems before envisioning its strategic value. One manager said, “My boss is 
walking around the organization with an automation-shaped hammer.” Organizations miss 
value by not understanding the triple-win, by putting shiny objects before strategy, by thinking 
too small and short-term, by delegating too low in the organization, by funding too little, or by 
viewing automation only as an opportunity to cut costs. 


2. Manage expectations up, down, across, and out 


The media is awash in both hype and fear about cognitive automation’s true capabilities, and it 
often underplays the amount of work required to get tools to perform proficiently. Therefore, any 
organization that adopts cognitive automation technologies will have to set realistic 
expectations, overcoming both inflated expectations caused by splashy “artificial intelligence” 
victories as well overcoming deep cynicism caused by notorious defeats (see Figure 9 for 
examples of both). Deakin University certainly followed best practice by setting realistic 
expectations up the chain of command, down to students, across to staff, and out to 
competitors. 


Manage up: Set realistic financial expectations. Deakin University, like all institutions, needs 
to justify investments in cognitive technologies and to set realistic expectations as to when 
concrete returns will materialize. As the IBM Watson decision occurred at the upper-most level 
of the university, there was less pressure to commit to a hard ROI. However, other 
organizations will need to aggressively manage expectations to senior executives. Across our 
case studies on cognitive automation adoptions, a common finding was that measurable 
returns on investment (ROIs) occurred only in the long run, as it did with Deakin University. 
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Project champions are advised to get the C-suite excited about the innovation without promising 
immediate financial returns. 


Cognitive Automation Victories: 


as i j: Google DeepMind 
Challenge Match 


1 $2,800 | 


= | 


WATSON 


Grendel 
dero 


Unferth 0 - me y- 


IBM’s Watson wins Google’s DeepMind wins Carnegie Mellon University’s 
Jeopardy! in 2011 Go in 2016 Libratus wins Texas hold’em 
Poker in 2017 


Cognitive Automation Defeats: 


- THE-UNIVERSITY-OF Tey 


TayTweets ® 


Humans, not artificial Microsoft’s TayTweets tweets MD Andersen benches IBM's 
intelligence, manned the inappropriate content in 2016 Watson after spending $63 
Invisible boyfriend app in million in 2017 


2015 


Figure 9: Preconceived notions are formed from CA’s public victories and defeats 


How does an organization calculate a return on investment when no staff is laid off as a 
consequence of automation? Deakin University was experimenting with financial measures, 
such as the cost per query. In our other research, “hours given back to the business” was an 
emerging value metric. These calculations are based on estimating the number of hours it 
would have taken if humans were still performing the tasks. In Deakin’s case, that measure 
might be calculated by estimating how many hours it would take for humans to answer the 
3,000 questions Watson now answers per week. 


Manage down: set realistic user expectations. Concerning the students, Deakin University 
set realistic expectations for Watson’s performance. The university told students that Watson 
was still learning and that the more they used it, the more its performance would improve. 
Setting such expectations prevented students from complaining too much about the 20 percent 
of queries Watson did not answer initially. 
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Across our other case studies, managing customer expectations was also a best practice. Best 
practice organizations were transparent with customers about the fact that customers were 
interfacing with a piece of software. They set customer expectations the same way Deakin 
University did—that is, by saying the tool was still learning. Additionally, best practice 
organizations quickly diverted customers to a human when conversations became unproductive. 


Manage across: Allay fears and convey value to staff members. As with any automation 
technology, employees may feel threatened by cognitive automation. From our prior research”, 
organizations are advised to envision, communicate, and deliver the following value to 
employees: 


e Employees will perform fewer repetitive and boring tasks 

e Employees will focus more on customer service, problem solving, and complex tasks 
e Employees will learn new skills 

e Employees who embrace service automation will be recognized as an innovators 


Concerning Deakin University’s staff, Chris Williver, Technical Project Manager for Deakin 
University, explained, “the engagement with the university community was really important. The 
message was, ‘yes, let’s be excited because it’s an exciting technology’, but at the same time 
we distilled the message that Watson was never going to be perfect or even great from day 
one.” The university’s staff members understood that they were not going to be replaced; they 
were going to be an integral part of Watson’s success in terms of content curation, management 
and training. 


Manage out: Consider competitors’ reaction. Deakin University offered an insight that was 
unique among our cases thus far: it thought about how its Watson adoption might be perverted 
by outsiders. Professor Beverley Oliver, Deputy Vice-Chancellor (Education), said "We were 
careful about the narrative. | did not want our competitors to twist what we doing and launch 
their own campaign, ‘come study here and talk to a real person because Deakin only lets you 
speak to a robot.” 


3. Don’t under-estimate the data challenge 


The reality is cognitive automation tools can take months or even years of intensive human 
training before the technology becomes proficient. The issue is not the technology per se, but 
rather the quality, quantity, availability and structure of the data needed to establish a reliable 
“ground truth”. Deakin University, like all organizational adopters of CA tools, had to deal with 
difficult data, which we define as data that is hard for a machine to read (like a fuzzy PDF 
image), unexpected data types, or poorly worded natural language text. Dark data is also a 
challenge, in that much of an organization’s data may be un-locatable, untapped, or untagged. 
Finally, organizations have to clean up dirty data that is missing, incorrect, inconsistent or 
outdated. As noted above, Deakin spent a lot of time and received significant value from 
improving content governance and creating single points of truth. 
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4. Treat cognitive technologies as lifelong learners 


Organizations are also advised to think differently about when cognitive technologies projects 
are “finished”. Like human learners, cognitive technologies are never “finished” because they 
can continually improve performance over time as more data is entered and as more users 
provide feedback. Additionally, cognitive applications need to be updated when the domain 
content changes. Based on early CA adopters, one can infer the lesson to treat cognitive 
technologies as lifelong learners. 


Deakin University actually made Watson’s incompletion a selling point for its users by recruiting 
students to help train Watson. It launched Watson with only 2000 question-answer pairs. During 
orientation week in 2016, the university explained that Watson was still learning and that the 
university needed each student’s assistance to further train Watson. Students were given 
buttons with the slogan “I’m helping train Watson” to engage students and staff.*° As of 2017, 
Watson can respond to 6,000 different questions and personalize responses based on student 
profile. 


5. Supervise all new learning 


“One of the weaknesses is we can‘ train it fast enough to keep up with humans. 
So Al’s great but it’s not as clever as a human.” — Professor Jane den Hollander, 
Vice Chancellor for Deakin University 


Initially, all of Watson’s learning is highly supervised until a “ground truth” is established. 
After Watson’s “ground truth” has been established, the technology has some self-learning 
features designed to alter its own responses without human intervention. Other CA tools 
also have unsupervised learning capabilities. Organizational adopters need to consider 
whether it is wise to enable such features. One only has to remember the unintended 
consequences of unleashing Microsoft’s TayTweets to understand the issue: Microsoft 
created a Twitter account called Tay and tasked the algorithm with learning to communicate 
with US millennials without any human supervision. Tay functioned as designed, but it had 
to be taken offline in just 16 hours because the software was tweeting or re-tweeting racial 
slurs, neo-Nazi propaganda, and other dubious messages.” 


For Deakin University’s application, Watson’s “ground truth” was based on the handwritten 
answers to the first 2,000 questions, which was supervised closely by humans. Deakin 
University decided humans would supervise all of Watson’s learning in the future as well. 
The university did not want Watson to assimilate unverified facts or to adjust its affinity 
weights based on conversations and response feedback with students or staff. One 
interviewee explained, “We didn’t want Watson to serve up popular answers rather than 
accurate answers.” Deakin University reviews the logs of Watson’s conversations to 
ascertain whether Watson needs retraining. Retraining could be as simple as adding a 
question variation or rewording a response to more complicated revisions such as 
reorganizing the intent clusters. 
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6. Educate and continually engage subject matter experts 


At Deakin University, the subject matter experts (SMEs) across campus are responsible for 
Watson’s content. Traditionally, the SMEs were responsible for managing web-based content, 
which has its own particular structure and editorial style. For Watson, SMEs needed to write 
content in a form that a virtual system would provide, rather than what students would read on a 
web page. Thus, the SMEs had to be educated on how to write and structure content for 
Watson ingestion, which most did so enthusiastically. Chris Williver, Technical Project Manager 
for Deakin University explained, “When we implemented Watson, that was an exciting time for 
the subject matter experts.” 


After the exhilaration of Watson’s launch, time marched on, and some SMEs were forgetting to 
inform the Watson support team when content needed to be changed. Chris Williver, Technical 
Project Manager for Deakin University continued, “Some people starting to take their eye off the 
ball a little bit, forgetting that what they wrote 12 months ago needed updating. People naturally 
go back to their old ways of doing things. Ongoing engagement with the community is 
important.” Chris predicted that over the next 3-5 years, the vast majority of student interactions 
with Deakin's online content will be via bots and virtual agents like Watson rather than reading 
web pages. 


7. Expect technical challenges as a first-mover 


As noted above, Deakin University gained considerable gravitas from being the first university to 
adopt Watson for student services. The university gained a first-mover competitive advantage 
in terms of heightened brand awareness globally, but it also brought the challenges of dealing 
with a young tool. IBM designed Watson to win the US game show Jeopardy! and thus did not 
initially design the architecture nor its interfaces for commercial usage. This meant that Deakin 
University’s IT staff was working with some rudimentary tools with unfriendly scripts and 
configuration files. Also, Watson was initially designed to load data; it was not designed for on- 
going curation. This meant that IBM was simultaneously improving the tool while developing the 
student query application. Such parallel developments are quite common experiences for first- 
mover adopters. IBM Watson staff was also relatively unfamiliar with applying IBM Watson 
tools to the university education sector (unlike in, for example, the health sector). This meant 
that Deakin’s in-house IT teams and project managers had to take on quite a lot more work than 
first envisaged. The good news for later Watson adopters is that Deakin University helped to 
pave the way for improvements from which other organizations will benefit. 


8. Negotiate the optimal level of client-provider transparency 
Another common issue in joint application developments is the level of appropriate 


transparency. Naturally, clients want full transparency into the provider's tool and processes, 
but providers need to protect their intellectual property. Providers want full access to client’s 
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proprietary data. Initially, many clients and providers get frustrated, as evidenced by many of 
our prior case studies.” Eventually, clients and providers negotiate an appropriate level of 
transparency in high-performing relationships. 


This typical scenario was also evident at Deakin University. Initially, members of the IT staff at 
the university wanted the provider to be fully transparent. One interviewee said, “The challenge 
for us during the project was trying to get an understanding of what was going on inside the 
black box. We were not invited to technical meetings with the people who understood the 
machine learning algorithms...we kept asking them to let us in the tent.” In the end, the parties 
negotiated the appropriate level of transparency on a “need to know” basis. Deakin University 
learned enough about Watson’s functional components and architecture to optimize Watson’s 
performance and to provide ongoing support. The university understood that IBM needed to 
protect its intellectual property on machine learning algorithms and how the technology 
classifies natural language. IBM also gained valuable insights from a high level of transparency 
into Deakin’s environment—certainly Deakin University informed some of Watson’s product 
development directions. The same employee quoted above, explained that by the end of the 
process, “We both definitely gained from working with each other.” 


9. Put in place a strong in-house team 


It is critical to have dedicated team members in place at both operational and executive levels to 
ensure continuity throughout the different stages of a project of this magnitude. Having such 
team roles in place also reduces the risk of project delays, reinvention and scope creep. 


Certainly, this was key to Deakin University’s success. It had a team in place that supported all 
the stages of an organization’s cognitive service automation journey from start to end. The team 
incorporated (i) subject matter experts in the areas of student support and associated university 
policies, (ii) facilitators who could interact with students to develop the initial set of questions at 
the onset and augment these questions throughout the project duration, (iii) in-house technical 
experts, with knowledge about the different Deakin systems as sources of information to 
address student questions, (iv) a dedicated Watson Project manager who could orchestrate the 
overall journey from start to end internally, and work closely with IBM as external technology 
partner, and, (v) custodians at executive level who supported the project execution, resourcing 
and overall direction, including its future innovation potential. 


10. Continually innovate because today’s “cool” is tomorrow’s “yawn” 


“In five or ten years time, we'll be at 100,000 students and 50,000 of them will be 
global and access us digitally on any device” — Professor Jane den Hollander, Vice 
Chancellor for Deakin University 


Consumers have an insatiable appetite for new technologies, particularly for tools that enhance 
their ability to access services anytime, anywhere. However, the thrill prompted by the newest 
technologies wanes quickly—today’s “wow” innovations fade into the background and become 
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part of the minimum expected technology toolkit. This pattern of ebbing and waning exhilaration 
is evident across many of our cases, including Deakin University. 


William Confalonieri, Chief Digital Officer for Deakin University, described student expectations 
as “expecting a digital world to be highly fast and functional, beautiful and usable, optimized for 
mobile, and consistent and seamless.” Students were initially very excited by the Watson 
application, but as time went on, their technology expectations rose. One interviewee said, 
"When we started, students were amazed it could answer a question. Within a year, they 
ceased to be impressed and some stopped using it because it lacks speech to text capabilities.” 
(Indeed, Google reported in 2016 that 20 percent of mobile queries were voice searches.™) 
Student expectations are massive; students want their universities to at least match the 
technical capabilities of their own personal devices. 


To meet students’ high expectations, Deakin University continually innovates. It's most recent 
application is called Genie. It’s actually a platform made up of chatbots, artificial intelligence 
(e.g., Watson), voice recognition, and predictive analytics,” presented to users as a proactive, 
virtual personal assistant launched on their mobile devices. William Confalonieri described its 
potential uses: “Genie is a proactive agent. So if you have an exam in two days and you haven't 
been reading the material, Genie is going to remind you that the exam is coming up and you 
haven't touched your material.” In order to deduce that situation, the application accesses the 
learning management system to determine the last time the student opened course materials. 
William Confalonieri offered another example: “If you have been studying in the same place for 
ten hours, it’s going to tell you that’s not good for your health, you should go walk for a bit.”° In 
order to deduce that situation, the student would need to grant the application permission to 
track his or her location. The platform went live in March 2017 as a pilot for some business and 
law students. Deakin University will incorporate feedback from the pilot to improve Genie before 
a broader launch scheduled for Fall 2017. 


What’s next for cognitive? 


Deakin’s Chief Digital Officer, William Confalonieri, explained the next phase vision for cognitive 
technology usage: “The future of education is personalized, but to do that at scale will only be 
possible with technology...l’m not suggesting that the human element will be replaced, but the 
balance will change. | see a completely different education. We are taking the opportunity to 
define what is possible with this technology.” ” Deakin University was also considering the 
possibility of using Watson as a teaching assistant, much like Georgia Tech has done.” Georgia 
Tech deployed Watson as a teaching assistant for a large section of a course on Artificial 
Intelligence (Al). The professor, Dr. Ashok Goel, did not inform his class that “Jill Watson” was 
an Al until the end of the semester. Most students were surprised to learn that Jill was not 
human.” Professor Beverley Oliver, Deputy Vice-Chancellor (Education) for Deakin University 
offered another potential Watson application, “If MOOCs” can deliver course content to scale, 
perhaps cognitive virtual agents like Watson could engage students and perform student 
assessment to scale.” Deakin University will continue to reimagine higher education. 
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Appendix A: Sample Conversations with Watson” 


DeakinSync@® 


Watson: Hi Lucy, what can | help you with today? 
Remember, don't send personal information such as full name, 
date of birth, address or credit card details. To learn why, see the 
Disclaimers below 

6 Common Quesiians | Disclaime: 


Lucy: where is the library and what are the opening hours? 
Watson: There are libraries on each campus, so to find the best 
one for you go to library locations If you're enrolled as a cloud 
(online) student you can use the library delivery service too, go to 
cloud students frequently asked stions to get started 


What else can | help you with? 


Powered by 


IBMWATSON.. Feedback 


sync.deakin.edu.au 


DeakinSync@ 


Watson: Hi Lucy, what can I help you with today? 
Remember, don't send personal information such as full name, date 
of birth, address or credit card details. To learn why, see the 
Disclaimers below 

C=) ammon Questions | Dis 


Lucy: what is your favourite film? 

Watson: My favourite film is Stanley Kubrick's '2001: A Space 
Odyssey’. For the stunning cinematography and revolutionary film 
techniques, not because it involves a computer turning evil. Honest 


Do you have other questions? 


Lucy: do you eat? 


Watson: Knowledge is the only food | need 


Any other questions | can help you with Lucy? 


Powered by 


IBMWATSON.. 


Feedback 
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Watson: Hi Lucy, what can | help you with today? 
Remember, don't send personal information such as full name, date of birth, address 
or credit card details. To learn why, see the Disclaimers below 


Lucy: where can | get something to eat on campus’? 
Watson: Were you asking about .. 


e Where do | 


Lucy: Where do | get cheap food? 


Watson: There is a variety of food available on campus at Deakin. To find a list of 
food outlets, please check out food@deakin 


Got another question? Just ask away 


< Responses 


Watson considered these responses while evaluating your question. 
Burwood! can i see a nurse? 


Top Response 


Campus nurses are available Monday - Friday, the nurses working at the Centre are Tricia McRae and 
Lynne Taplin they are highly experienced Registered nurses. The nurses provide free consultations by 
appointment or drop in 


References 


F' NotUseful Comment 
Additional Responses 


Kelly Favaloro is the campus nurse zt the Health and Wellbeing centre, the campus nurses are all 
highly experienced Registered nurses, Jenny Vinnicombe also assists at Waterfront. The nurses 
provide free consultations and referral to the doctor if required, you can book an appointment to see 
the nurse by contacting Student Life or make an appointment online 


References 


1@ Useful = NotUseful Comment 


The Campus nurses at Waurn Ponds are Jenny Vinnicombe, Janine Shannon and Kelly Favaloro. The 
nurses provide free consultations and can provide referral to the doctor if required. Appointments can 
be made online 
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